Camera surveillance planning and tracking system

ABSTRACT

Disclosed herein are system, method, and computer program product embodiments for camera surveillance planning and tracking using, for example, infrared technologies. An embodiment operates by distilling a moving foreground image from a frame, calculating a two-dimensional trajectory for the moving foreground object, determining a three-dimensional position for the moving foreground object, displaying the three-dimensional position on a map of an area.

BACKGROUND

Domestically, there are currently about 450 towered airports in theUnited States that do not have any significant surveillance capabilitiesto prevent potentially unsafe travel conditions. While there aresurveillance technologies available to these airports, such as radar(for example X-band radar), millimeter wavelength radar,multilateration, ADS-B capabilities, and virtual out-the-windowcapabilities based on camera technologies, these technologies areexpensive and cannot be economically justified by small to medium-sizedairports. Further, some of these technologies require equipage byvehicles and aircraft; thus, unequipped targets would be unseen by evenby some of these technologies. This leaves small to medium-sizedairports without any way to reliably detect and track vehicles andaircraft on the airport surface.

SUMMARY

Provided herein are system, apparatus, article of manufacture, methodand/or computer program product embodiments, and/or combinations andsub-combinations thereof, for camera surveillance planning and objecttracking.

An embodiment includes a computer implemented method for objectdetection and display. The method operates by distilling a movingforeground image from a frame, wherein the frame is received from acamera, calculating a two-dimensional trajectory for the movingforeground object based upon a feature of the moving foreground object,determining a three-dimensional position for the moving foregroundobject, wherein determining a three-dimensional position is based uponthe camera, and displaying the three-dimensional position for the movingforeground object on a map of an area.

Another embodiment includes a system for the construction of asurveillance plan. The system includes a memory and at least oneprocessor coupled to the memory. The processor is configured to distilla moving foreground image from a frame, wherein the frame is receivedfrom a camera, calculate a two-dimensional trajectory for the movingforeground object based upon a feature of the moving foreground object,determine a three-dimensional position for the moving foreground object,wherein determining a three-dimensional position is based upon thecamera, and display the three-dimensional position for the movingforeground object on a map of an area.

A further embodiment includes a non-transitory computer-readable devicehaving instructions stored thereon that, when executed by at least onecomputing device, cause the computing device to perform operations. Theoperations include distilling a moving foreground image from a frame,wherein the frame is received from a camera, calculating atwo-dimensional trajectory for the moving foreground object based upon afeature of the moving foreground object, determining a three-dimensionalposition for the moving foreground object, wherein determining athree-dimensional position is based upon the camera, and displaying thethree-dimensional position for the moving foreground object on a map ofan area.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a camera surveillance planning and trackingsystem 100, according to an example embodiment.

FIG. 2 is a diagram illustrating an example camera placement plannerunit 106 displaying example surveillance coverages A, B, C, and D (212,214, 216, and 218, respectively) of cameras A, B, C, and D (204, 206,208, and 210, respectively) on a map 200 of an area 202, according to anexample embodiment.

FIG. 3 is a diagram illustrating an example of camera placement plannerunit 106 displaying example surveillance coverages A, B, C, and D (312,314, 216, and 218, respectively) of cameras A, B, C, and D (304, 306,308, and 210, respectively) on a map 300 of an airport 302, according toa non-limiting example embodiment.

FIG. 4 is a flowchart illustrating a process for determining validstereo pairs in the surveillance plan 200 according to an embodiment.

FIG. 5 is a diagram illustrating an example of highlighting valid stereopairs on the map 500 of an area 502, according to an example embodiment.

FIG. 6 is a flowchart illustrating a process for determining an objectof interest from a continuous stream of video, according to an exampleembodiment.

FIG. 7 is a flowchart illustrating a process for computing 3D positionsfor display on GUI 114 according to an embodiment.

FIG. 8 is a diagram illustrating an example of GUI 114 displaying a mapseparated into blocks, according to an example embodiment.

FIG. 9 is an example computer system useful for implementing variousembodiments.

In the drawings, like reference numbers generally indicate identical orsimilar elements. Additionally, generally, the left-most digit(s) of areference number identifies the drawing in which the reference numberfirst appears.

DETAILED DESCRIPTION

Provided herein are system, method and/or computer program productembodiments, and/or combinations and sub-combinations thereof, forcamera surveillance planning and tracking. In a non-limiting embodiment,the camera surveillance planning and tracking system uses infraredtechnologies although other imaging technologies could alternatively beused—for example, spectral imaging, radar, sonar, ultraviolet, andvisible light technologies, to name a few.

FIG. 1 is a block diagram of a camera surveillance planning and trackingsystem 100. In an embodiment, camera surveillance planning and trackingsystem 100 constructs and visualizes a surveillance plan for an area bydisplaying surveillance coverages corresponding to a plurality ofcameras 102 according to inputs received.

In an embodiment, the camera surveillance planning and tracking system100 may include a plurality of cameras 102 that are installed accordingto the surveillance plan constructed by surveillance planning andtracking tool 104. The plurality of cameras 102 may transmit streams ofvideo to surveillance planning and tracking tool 104 to track andgeolocate objects from the streams of video. The camera surveillance andtracking tool 104 may comprise camera placement planner unit 106, camerasimulator unit 108, object detection unit 110, geolocation unit 112, andgraphic user interface (GUI) 114.

Although camera surveillance planning and tracking tool 104 isrepresented in FIG. 1 as a single physical machine, this is presented byway of example, and not limitation. In an additional embodiment, cameraplacement planner unit 106, camera simulator unit 108, object detectionunit 110, geolocation unit 112, and graphic user interface (GUI) 114 maybe made up of multiple physical machines in any combination thereof. Ina further embodiment, the plurality of cameras 102, camera placementplanner unit 106, camera simulator unit 108, object detection unit 110,geolocation unit 112, and GUI 114 communicate over a wired or wirelessnetwork.

In an embodiment, camera placement planner unit 106 constructs anddisplays a surveillance plan for an area according to a placement inputreceived. The surveillance plan may comprise displayed cameras andsurveillance coverages on the map of an area. Camera placement plannerunit 106 may receive a placement input corresponding to a camera of theplurality of cameras 102. The placement input may comprise a locationfor the camera, a specification input, and a pose for the camera, toname a few. Camera placement unit 106 may then display the camera on amap of an area based on the location for the camera.

According to an embodiment, camera placement planner unit 106 maycalculate and display the surveillance coverage, or field of view, forthe camera on a map of an area according to the placement inputreceived. Camera placement planner unit 106 may display the surveillancecoverage for the camera on a georeferenced plan view of an airport, as anon-limiting example.

According to an embodiment, camera placement planner unit 106 mayreceive a plurality of placement inputs corresponding to a plurality ofcameras 102. Camera placement planner unit 106 may construct asurveillance plan based on the plurality of placement inputs and displaythe plurality of cameras 102 and surveillance coverages for theplurality of cameras 102 on a map of an area according to the placementinputs.

In an embodiment, the plurality of cameras 102 may be installedaccording to the surveillance plan, i.e. how the plurality of cameras102 are displayed on the map of an area by camera placement planner unit106. The cameras 102 may undergo an intrinsic calibration procedure andan extrinsic calibration procedure. According to an embodiment, afterthe cameras undergo the intrinsic calibration procedure and extrinsiccalibration procedure, an updated specification input and updated posemay be calculated. The updated specification input and updated pose maybe sent to camera placement planner unit 106. This is described infurther detail below with reference to FIG. 2.

According to an embodiment, once the cameras 102 have been installed inthe area displayed by camera planner 106 according to the placementinputs, camera placement planner unit 106 may determine which of thecameras within the plurality of cameras 102 are valid stereo pairs, asdescribed below with reference to FIG. 3.

In an embodiment, camera simulator unit 108 may receive a simulationobject input. The simulation object input may comprise a 3D position onthe map displayed by camera placement planner unit 106. In anembodiment, the simulation object input simulates the 3D position of anobject of interest detected in the area displayed by the cameraplacement planner unit 106.

Camera simulator unit 108 may then use projective geometry and othertechniques to compute where the 3D simulation object input would appearon the image of the plurality of cameras specified in the surveillanceplan constructed and displayed in the camera placement planner unit 106.This computation produces in a plurality of 2D (pixel) locations, onefor each of the cameras 102 that can image the simulation object input.Camera simulator unit 108 may compile a plurality of object detectionpoints. The object detection points may comprise a unique cameraidentifier, 2D (pixel) location, and/or unique identifier of thesimulation object input. In an embodiment, camera simulator unit 108 mayreceive a plurality of simulation object inputs.

According to another embodiment, camera surveillance planning andtracking system 100 may track a plurality of objects from streams ofvideo transmitted by the plurality of cameras 102.

Object detection unit 110 may receive a continuous stream of video froma camera. The continuous stream of video may comprise video capturesfrom the surveillance areas of the cameras 102. Object detection unit110 may then use background subtraction and filtering to identify anobject of interest in the video, as described below with reference toFIG. 4. In an embodiment, object detection unit 110 may estimate anobject of interest's trajectory based upon the subtraction and filteringof the video.

In an embodiment, object detection unit may compile data related to theobject of interest, including locating information, identifyinginformation, and movement information, to name a few. Object detectionunit 110 may compile a plurality of object detection points based uponthe object of interest. The object detection points may comprise aunique camera identifier, 2D (pixel) location, and data regarding thesize, trajectory, and/or unique identifier of the object of interest.

In an embodiment, object detection unit 110 may receive a plurality ofcontinuous streams of video from the plurality of cameras 102.

According to an embodiment, geolocation unit 112 may receive a pluralityof object detection points from camera simulation unit 108 when asimulation object input is provided to the camera simulation unit 108.

According to an embodiment, geolocation unit 112 may receive a pluralityof object detection points from object detection unit 110 when theobjects of interest are detected across a set threshold of frames incontinuous video streams.

Geolocation unit 112 may then process the object detection points,received from the object detection unit 110 and/or the camera simulationunit 108, for any distortions and calculate a three-dimensional (3D)position—latitude, longitude, and altitude—from the processed objectdetection points using 3D triangulation for presentation in GUI 114, asdiscussed further in FIG. 6. 3D triangulation may comprise calculatingthe altitude of an object of interest by comparing the location of theobject of interest in frames of continuous video streams. In anembodiment, the frames are received from the cameras within a validstereo pair. According to an embodiment, geolocation unit 112 mayreceive a plurality of object detection points from detection unit 110and calculate a plurality of 3D positions based upon the plurality ofobject detection points.

In an embodiment, geolocation unit 112 may determine whether the 3Dposition is invalid, as discussed below with reference to FIG. 6.

According to an embodiment, geolocation unit 112 may comprise an objecttracker. The object tracker may maintain the continuity of the 3Dpositions calculated from the object detection points when objectdetection unit 110 cannot support the continuous presence of previouslydetected objects.

According to an embodiment, geolocation unit 112 may display thecomputed 3D positions on the map displayed by camera placement plannerunit 106. The geolocation unit 112 may highlight the valid stereo pairsof cameras that produced the object detection points used to perform thecomputation.

In an embodiment, GUI 114 may receive the 3D position from geolocationunit 112. GUI 114 displays an area divided into a plurality of blocks.According to an embodiment, the area displayed by GUI 114 may comprisethe area displayed by camera planning unit 106. In an embodiment, GUI114 may comprise a block area object tracker. When the 3D positionreceived comprises a position for an object of interest within one ofthe plurality of blocks, block area object tracker may highlight theblock corresponding to the object of interest's position, as discussedfurther in FIG. 7.

According to an embodiment, GUI 114 may receive a plurality of 3Dpositions. GUI 114 may highlight a plurality of blocks according to the3D positions received. GUI 114 may display the number of objects ofinterest within each highlighted block according to the 3D positions.

In an embodiment, GUI 114 may receive an updated 3D position for anobject of interest. From the updated 3D position, the block area objecttracker may calculate an averaged position, comprising a latitude,longitude, and altitude, for the object of interest. Using the averagedposition, block area object tracker may correct any discontinuities inthe blocks the object of interest moves between. Block area objecttracker may maintain the continuity of an object of interest whengeolocation unit 112 cannot support the continuous presence ofpreviously detected objects.

FIG. 2 is a diagram illustrating an example of camera placement plannerunit 106 displaying a surveillance plan 200 comprising surveillancecoverages A, B, C, and D (212, 214, 216, and 218, respectively) forcameras A, B, C, and D (204, 206, 208, and 210, respectively) on a mapof an area 202 constructed by camera placement planner unit 106,according to an embodiment.

In an embodiment, camera placement planner unit 106 receives a placementinput for a camera. The placement input may comprise a specificationinput and pose. The specification input may comprise attributes relatingto specifications of a camera, such as the focal length, field of view,and notional visual range, to name a few. According to an embodiment,the pose may comprise the camera x location, camera y location, camera zlocation, camera elevation angle, camera azimuth angle, and camera rollangle. According to an embodiment, the camera x location, camera ylocation, and camera z location may comprise locations on a map of anarea 202.

When camera placement planner unit 106 receives the placement input fora camera, the camera is added to the surveillance plan 200 and displayedon the map of an area 202 according to the placement input received. Forexample, camera placement 106 may receive a placement input for camera A204 comprising a certain camera x location, camera y location, andcamera z location that correspond to axes of the map of an area 202.Based upon these three locations, camera A 204 will be displayed on themap according to the camera x location, camera y location, and camera zlocation received.

According to an embodiment, camera placement planner unit 106 may alsocalculate a surveillance coverage for the camera. The surveillancecoverage may be calculated based upon the specification input and poseof the placement input for a camera. Once calculated, the surveillancecoverage may be added to the surveillance plan 200 and displayed on themap of the area 202.

For example, camera placement planner unit 106 may receive a placementinput comprising the pose for camera A 204 and a specification inputcomprising the manufacturer's specifications for the focal length, fieldof view, and notional visual range of camera A 204. According to theplacement input, camera placement plan 106 can calculate surveillancearea A 212, or field of view, of camera A 204. Camera placement plannerunit 106 can then add camera A 204 and surveillance area A 212 to thesurveillance plan 200 and display them on the map of the area 202.

In an embodiment, camera placement planner unit 106 may receive anupdated specification input when a camera undergoes an intrinsiccalibration process. The intrinsic calibration process may comprisecalibrating the intrinsic parameters of the camera—for example, thefocal length, image sensor format, lens distortion, and principal point,to name a few. The updated specification input may comprise a focallength, field of view, and notional visual range of a camera calculatedduring the intrinsic calibration process.

Camera placement planner unit 106 may update the surveillance plan 200with an updated surveillance coverage of a camera according to theupdated specification input. According to an embodiment, updating thesurveillance plan 200 with an updated surveillance coverage of a cameramay comprise calculating an updated surveillance coverage based on theupdated specification input, adding the updated surveillance coverage tothe surveillance plan 200, displaying the updated surveillance coverageof the camera on a map of an area 202, and removing the formersurveillance coverage of the camera from the surveillance plan 200.

For example, camera A 204 may undergo an intrinsic calibration where anupdated specification input comprising a focal length, field of view,and notional visual range is calculated. When camera placement plannerunit 106 receives the updated specification input, camera placementplanner unit 106 can calculate a new surveillance coverage for camera A204 based on the updated specification input. Camera placement plannerunit 106 may then update the surveillance plan 200 with the newsurveillance coverage by adding the new surveillance coverage for cameraA 204 to surveillance plan 200, displaying the updated surveillancecoverage for camera A 204 on the map of an area 202, and removingsurveillance coverage A 212 from surveillance plan 200.

According to an embodiment, camera placement planner unit 106 mayreceive an updated pose when a camera undergoes an extrinsic calibrationprocess. The extrinsic calibration process may comprise calibrating theextrinsic parameters of the camera—for example, the coordinate systemtransformation of the camera. The updated pose may comprise the updatedposition and orientation of a camera calculated during the extrinsiccalibration process. Camera placement planner unit 106 may then updatethe surveillance plan 200 with an updated surveillance coverageaccording to the updated pose.

For example, camera A 204 may undergo an extrinsic calibration process.During the extrinsic calibration process, an updated pose may becalculated for camera A 204 and sent to camera placement planner unit106. When camera placement planner unit 106 receives the updated pose,camera placement planner unit 106 can calculate a new surveillancecoverage for camera A 204 based on the updated pose. Camera placementplanner unit 106 may then update the surveillance plan 200 with the newsurveillance coverage by adding the new surveillance coverage for cameraA 204 to surveillance plan 200, displaying the updated surveillancecoverage for camera A 204 on the map of an area 202, and removingsurveillance coverage A 212 from surveillance plan 200.

In an embodiment, camera placement planner unit 106 may receive both anupdated specification input and updated pose and calculate an updatedsurveillance coverage for a camera according to the updatedspecification input and updated orientation. Camera placement plannerunit 106 may then update the surveillance plan 200 with the updatedsurveillance coverage of a camera.

FIG. 3 is a diagram illustrating an example of camera placement plannerunit 106 displaying example surveillance coverages A, B, C, and D (312,314, 216, and 218, respectively) of cameras A, B, C, and D (304, 306,308, and 210, respectively) on a map 300 of an airport runway 302,according to a non-limiting example embodiment.

FIG. 4 is a flowchart illustrating a process for determining validstereo pairs in the surveillance plan 200 according to an embodiment. Inan embodiment, a valid stereo pair may comprise a pair of cameras usedin 3D triangulation to determine the altitude of an object of interest.The pairs of cameras may comprise two cameras whose relationship to oneanother meets a preset criteria. For example, the criteria therelationship between the two cameras must meet may comprise the distancebetween the cameras, the overlap between the cameras' surveillancecoverages, the overlap between the functional ranges of view, and if theprincipal rays of the cameras subtend at a threshold angle, to name someexamples.

In an embodiment, the process for determining valid stereo pairs may beperformed by the camera placement planner unit 106 when a camera isadded to the system, removed from the system, or when an input placementis received.

According to an embodiment, at block 402, camera placement planner unit106 may calculate a set of stereo pairs based on all possible stereopairs in the surveillance plan 200. The set of stereo pairs may comprisepairs of the cameras and may be calculated in an n-choose-2(combinations without replacement) manner. For example, for cameras A204, B 206, C 208, and D 210, a set of stereos pairs comprising thepairs cameras A 204 and B 206, A 204 and C 208, A 204 and D 210, B 206and C 208, B 206 and D 210, and C 208 and D 210.

At block 404, camera placement planner unit 106 eliminates all camerapairs from the set of stereo pairs in which both cameras are collocated.For example, if cameras A 204 and B 206 are collocated, or at the samelocation, then the camera pair A 204 and B 206 would be eliminated fromthe set of stereo pairs.

At block 406, camera placement planner unit 106 eliminates all camerapairs in the set of stereo pairs in which there is no overlap in thesurveillance coverage of the two cameras. For example, surveillancecoverage 212 of camera A 204 may not overlap with surveillance overage218 of camera 210. Camera placement planner unit 106 would theneliminate the camera pair of A 204 and D 210 from the set of stereopairs.

At block 408, camera placement planner unit 106 eliminates all camerapairs from the set of stereo pairs in which the functional range of viewof the two cameras do not overlap. For example, the functional range ofview of camera B 206 and C 208 may not overlap. In this case, the camerapair of B 206 and C 208 would be eliminated from the set of stereopairs.

At block 410, camera placement planner unit 106 eliminates camera pairsfrom the set of stereo pairs in which the principal rays of the twocameras subtend at an angle that is less than a threshold angle. Forexample, the principal rays of cameras B 206 and D 210 may not subtendat an angle greater than a threshold angle. In this case, the camerapair of B 206 and D 210 would be eliminated from the set of stereopairs.

At block 412, the set of stereo pairs are considered a set of validstereo pairs and is sent to geolocation unit 112 for the computation of3D points.

FIG. 5 is a diagram illustrating an example of highlighting the validstereo pairs used to compute a 3D location on the map 500 of an area502, according to an embodiment.

In an embodiment, geolocation unit 112 may display the computed 3Dposition 504 on the map 500 of an area 502. Once the computed 3Dposition is displayed, geolocation unit 112 may highlight the validstereo pairs used to compute the corresponding 3D location. Highlightingthe valid stereo pairs used to compute the 3D position may comprisedisplaying lines (514 and 516) between the computed 3D position 504 andeach infrared camera that provided object detection points, through theobject detection unit 110, used to compute the 3D position.

For example, if the object detection unit 110 produced object detectionpoints for infrared camera pairs A 506 and B 508, geolocation unit 112subsequently computed 3D position 504 from these object detectionpoints. Geolocation unit 112 may display lines (514 and 516) between thecomputed 3D position 504 and infrared cameras A 506 and B 508 toindicate which valid stereo pairs resulted in the computed 3D position.

In an embodiment, geolocation unit 112 may display a plurality ofcomputed 3D positions and may highlight the valid stereo pairs for eachof the plurality of positions.

FIG. 6 is a flowchart illustrating a process for determining an objectof interest from a continuous stream of video frames, according to anembodiment.

At block 602, object detection unit 110 may receive a continuous streamof video from a camera. A frame of the received continuous stream ofvideo may then undergo a background subtraction. In an embodiment, thebackground subtraction may comprise a Gaussian Mixture Model-basedbackground subtraction.

At block 604, object detection unit 110 determines whether, after aframe of the continuous stream of video has undergone backgroundsubtraction, the frame displays one or more moving foreground objects. Amoving foreground object may comprise an area in a frame that differsfrom a base background image stored in the object detection unit 110 andis distilled from the frame when the base background image is subtractedfrom the frame. If one or more moving foreground objects are present inthe frame, the system continues to block 606. Otherwise block 602 isrepeated using the next frame of the continuous stream of video.

For example, object detection unit 110 may have a base background imagestored for camera A 204. When object detection unit 110 receives a framefrom the continuous stream of video of camera A 204, object detectionunit 110 would determine if any area of the frame from camera A 204differs from the stored base background image and is distilled from theframe when the stored base background image is subtracted from theframe.

According to an embodiment, the areas that differ from a base backgroundimage may be actual objects that are changing positions from frame toframe, or the result of noise in the camera data, as an example.

In an embodiment, object detection unit 110 may determine that the framedisplays a plurality of moving foreground objects.

At block 606, object detection unit 110 determines whether the movingforeground object can be associated with a foreground object identifiedin a previous frame and calculates a two-dimensional (2D) trajectorybased upon the object's movement between a plurality of frames.

According to an embodiment, at block 606 object detection unit 110performs optical flow tracking on the moving foreground object. Opticalflow tracking may comprise assigning a movement vector, or small 2Dtrajectory, to a moving foreground object based on an identifyingfeature of the moving foreground object. In an embodiment, theidentifying feature of the moving foreground object may comprise thebrightness, corners, or linear edges of a moving foreground object—toname a few. Object detection unit 110 may compare the location of theidentifying feature to the location of the identifying feature in aprevious frame. Based upon the difference in location, object detectionunit 110 may assign small 2D trajectory to the moving foreground object.

For example, the foreground object may comprise an identifying featurelocated at position X in a current frame at position Y in a previousframe. Based upon the identifying feature's movement between position Xand position Y, the object detection unit may calculate a 2D trajectoryfor the moving foreground object.

In an embodiment, a foreground object may comprise a plurality ofidentifying features. A 2D trajectory may be assigned to eachidentifying feature of the foreground object based on the identifyingfeatures' movements between frames. A 2D trajectory for the movingforeground object is calculated by averaging the 2D trajectories of eachidentifying feature together.

According to an embodiment, at block 606, object detection unit 110performs density-based clustering on a plurality of foreground objects.Density-based clustering may comprise comparing the 2D trajectories ofany of the plurality of foreground objects which are within a thresholddistance apart from each other. For example, comparing the 2Dtrajectories of foreground objects with are close together. If thedifference between the compared 2D trajectories is too great then themoving foreground objects are kept separate, otherwise they are groupedtogether as one moving foreground object.

At block 608, object detection unit 110 determines if the movingforeground object is an existing or new object of interest. In anembodiment, object detection unit 110 may perform Kalman filtering onobject detection points for objects of interest that are included on atracking list. An object of interest may comprise an airplane or vehicleon the surface of an airport, to name an example. Kalman filtering maycomprise calculating an estimated location and 2D trajectory for eachtracked object of interest according to the object detection points.Object detection unit 110 then compares the location and 2D trajectoryof the moving foreground object with the estimated locations and 2Dtrajectories of the tracked objects of interest. If the differencebetween the location and 2D trajectory of the moving foreground objectand the estimated location and 2D trajectory of a tracked object ofinterest is within a certain threshold, then the moving foregroundobject is determined to be an existing object of interest. When themoving foreground object is determined to be an existing object ofinterest, the system will continue to block 612.

If the difference between the location and 2D trajectory of the movingforeground object and all the estimated locations and 2D trajectories ofa tracked objects of interest is over a certain threshold, the movingforeground object is determined to be a new object of interest. The newobject of interest undergoes a threshold filtering. In an embodiment,the threshold filtering comprises a size and trajectory thresholdfiltering. For example, object detection unit 110 may determine if thesize and trajectory meet a certain threshold. If the new object ofinterest does not meet the threshold for size and trajectory the systemwill continue on to block 610, otherwise the system will continue on toblock 612.

At block 610, the new object of interest is discarded. The system thencontinues to block 618.

At block 612, object detection unit 110 adds a new object detectionpoint for a newly designated object of interest or an existing object ofinterest to the tracking list. The object detection point, may compriseof the location, size, 2D trajectory, and unique identifier of theobject of interest, to name a few. The system then proceeds to block614.

At block 614, object detection unit 110 determines whether the object ofinterest has been persistently identified, with consistent location,size, and 2D trajectory attributes in a threshold number of frames. Ifthe object of interest has been persistently tracked in a thresholdnumber of frames, the system will continue to block 616. Otherwise, thesystem proceeds to block 618.

At block 616, the object of interest is included in the update list thatis continuously sent to geolocation unit 112. According to anembodiment, the object of interest is removed from the update list whenthe object of interest has not been persistently identified in athreshold number of frames. The system then continues to block 618.

At block 618, object detection unit 110 determines if there has been adetermination whether each of the plurality of moving foreground objectsdisplayed by the frame is an object of interest or not. If adetermination has not been made for each of the plurality of movingforeground objects displayed in the frame, then block 608 is repeatedfor a different moving foreground object from the plurality of movingforeground objects. Otherwise, block 602 is repeated using the nextframe of the continuous stream video.

FIG. 7 is a flowchart illustrating a process for computing 3D positionsfor display on GUI 114 according to an embodiment.

At block 702, geolocation unit 112 receives an object detection pointand corrects the 2D location of the object detection point for anydistortions produced by the camera's lens. In an embodiment, thecorrection of the 2D location of the object detection point may beperformed according to the updated specification input.

At block 704, geolocation unit 112 may calculate all possible 3Dpositions the corrected object detection points. The 3D positions maycomprise a latitude, a longitude, and an altitude. Calculating the 3Dposition may comprise 3D triangulation. Geolocation unit 112 may use theobject detection points produced by the object detection unit 110 tocalculate a 3D position.

At block 706, geolocation unit 112 determines if the 3D position for theobject of interest comprises an invalid point. An invalid point maycomprise a 3D position that comprises a negative altitude, a locationoutside of an area of interest, or a location historically associatedwith false targets, to name a few. For example, a 3D position on ahighway outside the perimeter of an airport would be outside the area ofinterest and be determined to be an invalid point. All such points areremoved from the set of computed 3D positions computed in block 704.

At block 708, geolocation unit 112 adds the remaining computed 3D pointsto the object tracker. The object tracker may maintain the continuity ofthe 3D positions calculated from the object detection points when objectdetection unit 110 cannot support the continuous presence of previouslydetected objects.

At block 710, geolocation unit 112 sends all 3D positions maintained inthe object tracker, if any, to GUI 114.

FIG. 8 is a diagram illustrating an example of GUI 114 displaying a mapseparated into blocks, according to an embodiment.

GUI 114 may display a map 800 of an area 802 separated into blocks. Theblocks define and encompass discrete areas that the user requiressurveillance. The blocks may or may not be contiguous. For example, themap 800 of an area 802 may be separated into blocks A, B, C, D, E, and Fblocks (804, 806, 808, 810, 812, and 814, respectively). In anembodiment, individual blocks may vary in size (length, width and shape)from other blocks. GUI 114 may receive and track 3D position updates foreach object of interest identified by the system. The 3D positionupdates may comprise the longitudes, latitudes, and altitudes of anobject of interest calculated from a plurality of frames from acontinuous video stream. GUI 114 may average the tracked 3D positionupdates to calculate a single longitude, latitude, and altitude for theobject of interest.

In an embodiment, GUI 114 may average the tracked 3D position updates tocalculate a plurality of single longitude, latitude, and altitude for aplurality of objects of interest.

According to an embodiment, GUI 114 may then determine whether a singlelongitude, latitude, and altitude of the object of interest is withinone of the blocks (804, 806, 808, 810, 812, and 814) that map 800 of anarea 802 is separated into. For example, once GUI 114 calculates asingle longitude, latitude, and altitude of an object of interest, GUI114 may determine that the single longitude, latitude, and altitude arewithin block E 812.

In an embodiment, when a single longitude, latitude, and altitude of anobject of interest is in a block on the map, GUI 114 may highlight theblock the single longitude, latitude, and altitude are in. For example,if the single longitude, latitude, and altitude are within block E 812,then GUI 114 would highlight block D 812 on the map 800 of the area 802.

According to an embodiment, GUI 114 may determine that a plurality ofsingle longitude, latitude, and altitude of a plurality of objects ofinterest are within a block on the map. When a plurality of singlelongitudes, latitudes, and altitudes of a plurality of objects ofinterests are within a block, GUI 114 may highlight a block and displaythe number of objects of interests within the block.

Various embodiments can be implemented, for example, using one or morewell-known computer systems, such as computer system 900 shown in FIG.8. Computer system 900 can be any well-known computer capable ofperforming the functions described herein.

Computer system 900 includes one or more processors (also called centralprocessing units, or CPUs), such as a processor 904. Processor 904 isconnected to a communication infrastructure or bus 906.

One or more processors 904 may each be a graphics processing unit (GPU).In an embodiment, a GPU is a processor that is a specialized electroniccircuit designed to process mathematically intensive applications. TheGPU may have a parallel structure that is efficient for parallelprocessing of large blocks of data, such as mathematically intensivedata common to computer graphics applications, images, videos, etc.

Computer system 900 also includes user input/output device(s) 903, suchas monitors, keyboards, pointing devices, etc., that communicate withcommunication infrastructure 906 through user input/output interface(s)902.

Computer system 900 also includes a main or primary memory 908, such asrandom access memory (RAM). Main memory 908 may include one or morelevels of cache. Main memory 908 has stored therein control logic (i.e.,computer software) and/or data.

Computer system 900 may also include one or more secondary storagedevices or memory 910. Secondary memory 910 may include, for example, ahard disk drive 912 and/or a removable storage device or drive 914.Removable storage drive 914 may be a floppy disk drive, a magnetic tapedrive, a compact disk drive, an optical storage device, tape backupdevice, and/or any other storage device/drive.

Removable storage drive 914 may interact with a removable storage unit918. Removable storage unit 918 includes a computer usable or readablestorage device having stored thereon computer software (control logic)and/or data. Removable storage unit 918 may be a floppy disk, magnetictape, compact disk, DVD, optical storage disk, and/any other computerdata storage device. Removable storage drive 914 reads from and/orwrites to removable storage unit 918 in a well-known manner.

According to an exemplary embodiment, secondary memory 910 may includeother means, instrumentalities or other approaches for allowing computerprograms and/or other instructions and/or data to be accessed bycomputer system 900. Such means, instrumentalities or other approachesmay include, for example, a removable storage unit 922 and an interface920. Examples of the removable storage unit 922 and the interface 920may include a program cartridge and cartridge interface (such as thatfound in video game devices), a removable memory chip (such as an EPROMor PROM) and associated socket, a memory stick and USB port, a memorycard and associated memory card slot, and/or any other removable storageunit and associated interface.

Computer system 900 may further include a communication or networkinterface 924. Communication interface 924 enables computer system 900to communicate and interact with any combination of remote devices,remote networks, remote entities, etc. (individually and collectivelyreferenced by reference number 928). For example, communicationinterface 924 may allow computer system 900 to communicate with remotedevices 928 over communications path 926, which may be wired and/orwireless, and which may include any combination of LANs, WANs, theInternet, etc. Control logic and/or data may be transmitted to and fromcomputer system 900 via communication path 926.

In an embodiment, a tangible apparatus or article of manufacturecomprising a tangible computer useable or readable medium having controllogic (software) stored thereon is also referred to herein as a computerprogram product or program storage device. This includes, but is notlimited to, computer system 900, main memory 908, secondary memory 910,and removable storage units 918 and 922, as well as tangible articles ofmanufacture embodying any combination of the foregoing. Such controllogic, when executed by one or more data processing devices (such ascomputer system 900), causes such data processing devices to operate asdescribed herein.

Based on the teachings contained in this disclosure, it will be apparentto persons skilled in the relevant art(s) how to make and useembodiments of the invention using data processing devices, computersystems and/or computer architectures other than that shown in FIG. 8.In particular, embodiments may operate with software, hardware, and/oroperating system implementations other than those described herein.

It is to be appreciated that the Detailed Description section, and notthe Summary and Abstract sections (if any), is intended to be used tointerpret the claims. The Summary and Abstract sections (if any) may setforth one or more but not all exemplary embodiments of the invention ascontemplated by the inventor(s), and thus, are not intended to limit theinvention or the appended claims in any way.

While the invention has been described herein with reference toexemplary embodiments for exemplary fields and applications, it shouldbe understood that the invention is not limited thereto. Otherembodiments and modifications thereto are possible, and are within thescope and spirit of the invention. For example, and without limiting thegenerality of this paragraph, embodiments are not limited to thesoftware, hardware, firmware, and/or entities illustrated in the figuresand/or described herein. Further, embodiments (whether or not explicitlydescribed herein) have significant utility to fields and applicationsbeyond the examples described herein.

Embodiments have been described herein with the aid of functionalbuilding blocks illustrating the implementation of specified functionsand relationships thereof. The boundaries of these functional buildingblocks have been arbitrarily defined herein for the convenience of thedescription. Alternate boundaries can be defined as long as thespecified functions and relationships (or equivalents thereof) areappropriately performed. Also, alternative embodiments may performfunctional blocks, blocks, operations, methods, etc. using orderingsdifferent than those described herein.

References herein to “one embodiment,” “an embodiment,” “an exampleembodiment,” or similar phrases, indicate that the embodiment describedmay include a particular feature, structure, or characteristic, butevery embodiment may not necessarily include the particular feature,structure, or characteristic. Moreover, such phrases are not necessarilyreferring to the same embodiment. Further, when a particular feature,structure, or characteristic is described in connection with anembodiment, it would be within the knowledge of persons skilled in therelevant art(s) to incorporate such feature, structure, orcharacteristic into other embodiments whether or not explicitlymentioned or described herein.

The breadth and scope of the invention should not be limited by any ofthe above-described exemplary embodiments, but should be defined only inaccordance with the following claims and their equivalents.

What is claimed is:
 1. A method, comprising: distilling a movingforeground image from a frame, wherein the frame is received from acamera; calculating a two-dimensional trajectory for the movingforeground image based upon a feature of the moving foreground object;determining a three-dimensional position for the moving foregroundobject, wherein determining a three-dimensional position is based uponthe camera; and displaying the three-dimensional position for the movingforeground object on a map of an area.
 2. The method of claim 1, whereinthe moving foreground image comprises a plurality of features.
 3. Themethod of claim 2, the calculating further comprising: assigning avector to each of the plurality of features.
 4. The method of claim 3,wherein the two-dimensional trajectory comprises a mean of the vectors.5. The method of claim 1, wherein the camera comprises one of a pair ofstereo cameras.
 6. The method of claim 5, the determining athree-dimensional position, wherein determining the three-dimensionalposition is further based upon a relationship between the pair of stereocameras.
 7. The method of claim 1, wherein the map of an area comprisesa plurality of blocks, wherein each block comprises a section of thearea.
 8. The method of claim 7, the displaying further comprising:highlighting at least one of the plurality of blocks according to thethree-dimensional position.
 9. A system, comprising: a memory; and atleast one processor coupled to the memory and configured to: distill amoving foreground image from a frame, wherein the frame is received froma camera; calculate a two-dimensional trajectory for the movingforeground image based upon a feature of the moving foreground object;determine a three-dimensional position for the moving foreground object,wherein determining a three-dimensional position is based upon thecamera; and display the three-dimensional position for the movingforeground object on a map of an area.
 10. The system of claim 9,wherein the moving foreground image comprises a plurality of features.11. The system of claim 10, the at least one processor configured tocalculate, further configured to: assign a vector to each of theplurality of features.
 12. The system of claim 9, wherein the cameracomprises one of a pair of stereo cameras.
 13. The system of claim 12,the at least one processor configured to determine, wherein determiningthe three-dimensional position is further based upon a relationshipbetween the pair of stereo cameras.
 14. The system of claim 9, whereinthe map of an area comprises a plurality of blocks, wherein each blockcomprises a section of the area.
 15. A tangible, non-transitorycomputer-readable device having instructions stored thereon that, whenexecuted by at least one computing device, causes the at least onecomputing device to perform operations comprising: distilling a movingforeground image from a frame, wherein the frame is received from acamera; calculating a two-dimensional trajectory for the movingforeground image based upon a feature of the moving foreground object;determining a three-dimensional position for the moving foregroundobject, wherein determining a three-dimensional position is based uponthe camera; and displaying the three-dimensional position for the movingforeground object on a map of an area.
 16. The non-transitorycomputer-readable device of claim 15, wherein the moving foregroundimage comprises a plurality of features.
 17. The non-transitorycomputer-readable device of claim 16, the calculating operation furthercomprising: assigning a vector to each of the plurality of features. 18.The non-transitory computer-readable device of claim 17, wherein thetwo-dimensional trajectory comprises a mean of the vectors.
 19. Thenon-transitory computer-readable device of claim 18, wherein the cameracomprises one of a pair of stereo cameras.
 20. The non-transitorycomputer-readable device of claim 19, the determining operation, whereindetermining the three-dimensional position is further based upon arelationship between the pair of stereo cameras.